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Comparing the Performance of Allopathically and Osteopathically Trained Physicians 
on the American Board of Family Medicine’s Certification Examination 
Two medical specialty boards offer certification in family medicine for physicians: the 
American Board of Family Medicine (ABFM), under the auspices of the American Board of 
Medical Specialties; and the American Osteopathic Board of Family Physicians (AOBFP), under 
the auspices of the American Osteopathic Association (AOA). The AOBFP certification is 
offered only to graduates of osteopathic colleges; however, graduates of both osteopathic and 
allopathic medical schools who have successfully completed a residency program accredited by 
the Accreditation Council for Graduate Medical Education (ACGME) may seek certification by 
the ABFM. Some allopathic family medicine residency programs are accredited by both the 
ACGME and the AOA, which qualifies their residents to seek certification from either or both 
certification boards. Such programs are referred to as having “dual accreditation.” 

The performance of the following are addressed herein: (1) first-time candidates on the 
ABFM certification examination, (2) candidates from allopathic and osteopathic colleges, and 
(3) candidates from allopathic and osteopathic colleges who completed dually accredited 
residency programs. Issues of attrition are also addressed. 

Method 

This study used a natural groups design with a matching variable in some cases to test for 
differences in performance on the ABFM certification examination between osteopathically and 
allopathically trained physicians. 

Participants 

The participants in this study were first-time candidates for ABFM certification who 
tested in 2007 and 2008. Only a candidate’s first attempt on the examination was considered. All 
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of these candidates graduated from a college of medicine, held an unrestricted U.S. medical 
license, and successfully completed an ACGME accredited residency program. A subset of this 
cohort consisted only of the candidates who successfully completed a residency program that 
was accredited by both ACGME and AOA. 

Instrument 

The instrument employed in this study was the ABFM’s Certification Examination. This 
examination consisted of 370 multiple choice items that are dichotomously scored. However, 20 
of the items are pretest items, resulting in 350 items that contributed to the candidate’s score. The 
test was designed in accordance with the test blueprint. 1 Additional information about the test 
can be found in the Candidate Information Booklet. 2 Scoring of the test was performed using the 
dichotomous one parameter logistic (1PL) model, also known as the Rasch measurement model. 3 
Analysis 

MD/DO Comparison. 

Using an independent samples t-test, the scaled scores for MDs and DOs were compared twice, 
first considering the entire cohort and second considering only the subset from the dually 
accredited programs. Pass rates for these groups were then compared using an independent 
samples test for differences between the two percentages. The calculation of the standard error of 
the pass rate was defined as: 




where, 

s p = 

P 

q 

N 



the standard error of the proportion 
pass rate 
1 -P 

sample size 
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The following formula was used to transform the difference between the percentages into a z- 



score: 



£ ■ 




where, 



Pdo - 

2 

S p MD — 
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S p DO — 



Pmd 



the standardized z score 

the pass rate for MDs 

the pass rate for DOs 

the variance for the MD pass rate 

the variance for the DO pass rate 



Delay and Attrition. 

To determine any decline in the number of eligible residents taking the certification 
examination after Program Year 3 (PGY3), counts were provided for the same cohort beginning 
in 2004 and ending in 2007, as residents progressed through their residency as indicated by 
Program Year 1 (PGY1), Program Year 2 (PGY2), PGY3, and finally those taking the 
certification exam in 2007. The average number of candidates at each point in time was 
determined and compared. 



MD and DO Performance Comparisons on the ABFM Examination 

In 2007 and 2008 respectively, there were 3,329 and 3,111 first-time candidates 
attempting to earn their initial certification (Table 1). Most of the candidates (86% and 87% 
respectively) were MDs, with the remainder being DOs. 

In 2007, MDs had a greater propensity to pass ( M = 79.8%, SS Basa Satv = 0.0075) than 
DOs (M = 74.6%, SEgaas Katv = 0.0202), and a difference of this magnitude is unlikely to have 
occurred by chance, t (599.3) = -2.38, p < .018. In 2008, MDs (Af = 83.8%, SD = 32.2) and DOs 



Results 
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(M = 83.2%, SD = 33.1) had pass rates that were not significantly different, t (1944) = 0.38, p < 
.708. 



The mean scaled scores for these groups were computed and subsequently tested for 
statistically significant differences. The unequal variance formula for t-tests was employed 
because Levene’s test for homogeneity of variance yielded a significant difference between the 
variances of the MD and DO groups. For 2007, F (1,3327) = 7.72, p < .005; for 2008, F (1,3109) 
= 5.17, p < .023. In 2007, MDs scored higher (M = 457.6, SD = 85.4) than DOs (M = 437.6, SD 
= 78.7) and a difference of this magnitude is unlikely to have occurred by chance, t (654.5) = 
5.02, p < .001. In 2008, MDs again scored higher ( M = 473.1, SD = 85.7) than DOs (M = 459.8, 
SD = 78.9) and a difference of this magnitude is also unlikely to have occurred by chance, t 
(536.7) = 3.077, p < .002. 



Table 1. Summary of Performance by ABFM Candidates for Initial 
Certification in 2007 and 2008 





2007 




2008 




MD 


DO 


MD 


DO 


Count 


2864 


465 


2717 


394 


Passing 


2284 


347 


2277 


328 


Pass Rate 


* 79.7% 


*74.6% 


83.8% 


83.2% 


Mean Score 


* 457.6 


*437.6 


*473.1 


*459.8 



*Indicates that there was a statistically significant difference between MDs and DOs at the a < 



.05 level. 

Dually Accredited Programs 

According to the American Osteopathic Association, as of 2009, 91 residency programs 
were accredited by both the ACGME and the AOA. 4 Of these 91 programs, only 86 had valid 
ACGME Program ID numbers listed on the AOA Website. The five programs without ACGME 
ID numbers were excluded from this study. 
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The following analysis was based on the scores of candidates who completed a dually 
accredited residency program. It should be noted that only candidates’ first attempt on the initial 
certification examination were included. Therefore, candidates who had previously failed or who 
were testing to become recertified were excluded from the analysis. 

MD and DO Comparisons among Dually Accredited Programs on the ABFM Examination 

With regard to whether the MDs or the DOs from dually accredited programs had a 
greater propensity to pass, the results were mixed (Table 2). In 2007, MDs (M = 77.4%, 
^BassBa r*= 0.0188) passed at a higher rate than DOs (M = 72.7%, SE Es3S = 0.0475); 
although, the results were not statistically significant, t (582) = -0.96, p < .338. Similarly, the 
difference in mean scaled scores between MDs (M = 450.2, SD = 81.7) and DOs (M = 448.5, SD 
= 87.7) was small and well within boundaries of random variation, t (582) = -0.18, p < .857. In 
2008, DOs (M = 84.6%, SD = 33.5) slightly outperformed MDs (M = 83.9%, SD = 33.2) with 
regard to pass rates, but MDs (M = 472.0, SD = 84.7) attained higher mean scaled scores than 
DOs (M = 461.8, SD = 77.4). This is likely due to the effects of a small sample of DOs (n = 65) 
compared to a larger and more densely populated distribution of MDs (n = 453). The difference 
in mean scaled scores between MDs and DOs, however, was small and within boundaries of 
random variation, t (317) = 1.83, p < 0.068, so it is likely these results occurred by chance. 



Table 2. Summary of Performance by ABFM Candidates from Dually 
Accredited Residency Programs for Initial Certification in 2007 and 2008 





2007 




2008 






MD 


DO 


MD 


DO 


Count 


496 


88 


453 


65 


Passing 


384 


64 


380 


55 


Pass Rate 


77.4% 


72.7% 


83.9% 


84.6% 


Mean Score 


450.2 


448.5 


472.0 


461.8 
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*There were no statistically significant difference between MD and DO cohorts at the a < .05 
level. 

Delay and Attrition 

It is reasonable to assume that the directors of the dually accredited residency program 
want all of their program’s residents to both sit for and pass the ABFM certification examination. 
However, some DOs may decide to forego ABFM certification in favor of osteopathic 
certification, which raises this question: “Among dually accredited programs, is there a sizeable 
cohort of DOs that complete the residency training, but that do not apply for ABFM certification 
the following year?” 

To investigate this question, the records for the dually accredited programs were queried 
from ABFM’s Resident Training Management System (RTMS) to identify the number of 
residents from each of the programs that took the residents’ In-Training Examination (ITE). It is 
believed that nearly all program directors require their residents, regardless of the type of 
medical school they attended, to take the ITE. Therefore, the difference between the mean 
number of candidates taking the ITE and the mean number of first-time candidates taking the 
examination the following year should be a good indicator of how many residents typically opt 
not to take the examination. The results suggest that the average program has 7 residents but only 
6 who sit for the examination (Figure 1). 



Page 7 of 16 




2004 Resident Cohort for Dually Accredited Programs 




PGY12004 PGY2 2005 PGY3 2006 CERT 2007 



Figure 1. Mean number of residents taking the ITE and ABFM certification 
examinations. 

An investigation into resident enrollment and attrition among various programs from the 
PGY3 to the 2007 Certification year reveals interesting findings. Figure 2 displays the frequency 
distribution of these change rates. Programs with no changes (n = 22) are the most prevalent, but 
programs that lost 1 (n = 21) or 2 (n = 17) residents closely follow. Twelve programs lost 3 or 
more residents during this timeframe, while 11 programs actually reported gaining 1 to 2 
residents. It is likely the instances of increase are attributable to residents who were eligible to 
complete the examination in 2006, but opted to postpone the initial attempt at the examination 
until the following year. 
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25 




Changes 

Figure 2. Difference between 2006 PGY3 residents and 2007 ABFM 
examinees. 

Discussion 



Issues of Comparison 

There are at least two factors that make the MD and DO performance comparison less 
clear: selection bias and comparability of constructs. Because there are two medical specialty 
boards that offer certification in family medicine, only DOs have the option of pursuing either or 
both AOBFP and ABFM certifications. Because MDs cannot obtain AOBFP certification, any 
comparative analyses between examinees’ successes and failures with regard to the two 
certification boards’ examinations are limited for dominantly two methodological reasons. The 
first pertains to the aforementioned problem of true comparative data, as one cannot observe 
performance on matched cohorts because MDs cannot sit for the AOBFP examination. The 
second pertains to the issue of comparability between constructs. Ideally, one would need to 
know if these tests generally measure the same construct. Assessing the comparability of the 
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construct is largely a content-related endeavor, and we are unaware of any literature that 
addresses ABFM and AOBFP examination similarities or dissimilarities. 

Bias in Selecting a Program 

Among medical graduates there are noticeable individual differences in ability. 
Similarly, there are differences in the quality of residency programs as perceived by the pool of 
new residents. Therefore, medical graduates span a continuum of ability and residency programs 
span a range of quality. If residents seek to get into the best program that will accept them and 
residency programs accept only the best applicants to their program, then there is likely to be a 
positive correlation between the quality of resident and the quality of the program. This may 
cause differences in examination performance between MD and DO cohorts to be smaller when 
only the dually accredited programs are considered, because the quality of the programs are held 
constant. 

Sources of Attrition 

On average, there were approximately 7 persons enrolled in each dually accredited PGY3 
cohort in 2006. The average number of candidates from these cohorts who elected to take the 
ABFM certification exam was approximately 6. These data indicate about 1 person per program 
within the cohort opts not to sit for the ABFM certification examination each year. 

In 2008, the number of DOs seeking ABFM initial certification (n = 394) decreased by 71 
from the 2007 year (n = 465). The number of DO residents from dually accredited residency 
programs seeking ABFM initial certification also dropped from 88 to 65 during the 2007 and 
2008 years, a decrease of 23. Analyses of these data consistently suggest a small number of 
residents, some of whom are from dually accredited programs, do not seek ABFM certification. 
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In 2006, there was no unique common identifier used to connect a candidate’s residency 
records to his or her records related to certification. Because the demographic information is 
largely part of the candidate records, identifying the demographics for people who did not test is 
not possible. Because of this limitation the analyses did not explore the demographics related to 
medical training, MD and DO, therefore it is unclear whether the missing cases consist of MDs, 
DOs, or both. Also, the inability to differentiate between actual instances of attrition and mere 
deferrals to taking test further limits this study. Given there is some degree of attrition or 
deferral, the authors provide several plausible reasons why residents might not have taken the 
ABFM’s Certification Examination. 

First, the resident may have taken a job in another city or state and the idea of taking on a 
new job, moving, and preparing for the ABFM examination may seem to be too much to take on 
all at once. These physicians may have decided to delay taking the examination. Other plausible 
reasons for delaying taking the examination could include having a child, dealing with a serious 
illness, or resolving a family crisis. 

Second, some residents may have opted to become exclusively ABOFP certified. There 
are several reasons for why they might make this choice. One is that AOA requires 90% of 
residents within their accredited residency programs to take the ABOFP examination, which has 
a very high pass rate and the test results are available before the deadline to apply for the ABFM 
examination. 5 Some residents may consider the additional certification to be too time consuming 
and expensive to pursue. 

Summary 

When comparing the performance of residents from ACGME accredited family medicine 
residency programs on the ABFM’s Certification Examination, there is a statistically significant 
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difference between residents who graduated from allopathic medical schools and osteopathic 
medical schools. With regard to the scaled scores, the allopathically trained physicians 
outperformed the osteopathically trained physicians by 13 to 20 points. Although this difference 
was larger than what could be attributed to chance, it did not always translate to a difference in 
pass rate that was statistically significant. After controlling for differences in the quality and 
selectivity of the residency programs by considering only the dually accredited residency 
programs, the statistical differences vanished. The data also suggested that for the dually 
accredited programs the average class size was 7 residents, but only 6 take the test the following 
year. It was unclear how much of this decrease was attrition and how much was a delay in 
testing. 



Page 12 of 16 




References 



1 . American Board of Family Medicine. Certification/Recertification Examination Content. 
American Board of Family Medicine, 

https://www.theabfm.org/cert/CertRecertExaminationOutline.pdf, accessed June 17, 2009. 

2. American Board of Family Medicine. Candidate Information Booklet. American Board of 
Family Medicine, https://www.theabfm.org/cert/candidateinfobooklets.aspx, accessed June 
17, 2009. 

3. Rasch G. Probabilistic models for some intelligence and attainment tests. Copenhagen, 1960. 

4. American Osteopathic Association. Welcome to Opportunities — AOA-Approved Internships 
and Residencies. American Osteopathic Association, 

http://opportunities.osteopathic.org/search/search.cfm, accessed June 15, 2009. 

5. Terry R. Dual Accreditation: Certification Reality. Presentation for Wilson Memorial 
Hospital, http://www.do-online.org/pdf/20090ME-PPT/Terry%20- 
%20Dual%20programs%202009.ppt, accessed June 16, 2009. 



Page 13 of 16 




Appendix A. Residency Programs that are Accredited by Both ACGME and AOA. 



SEQ 


AOA# 


ACGME# 


State Family Practice/Family Medicine Residency Program 


01 


183204 


1200221596 


AK 


PCSOM/Alaska Family Medicine Residency 

Providence-Family Practice Residency 


02 


126035 


1200411037 


CA 


WesternU/COMP Arrowhead Regional Medical Center 


03 


136635 


1200411037 


AR 


OSUCOM/UAMS-AHEC 


04 


130876 


1200911079 


DE 


PCOM/Christiana Care Health Services 


05 


169601 


1201111085 


FL 


NSUCOM/St Vincents Medical Center 


06 


169593 


1201211092 


GA 


NSUCOM/The Medical Center 


07 


163815 


1201221091 


GA 


NSUCOM/Medical College of GA 


08 


180852 


1201231094 


GA 


NSUCOM/Floyd Medical Center 


09 


148262 


1201611098 


IL 


MWU/CCOM/MacNeal Hospital 


10 


161923 


1201611099 


IL 


KCOM/SIU/Memorial Hospital of Carbondale 


11 


169600 


1201611102 


IL 


Resurrection Medical Center 


12 


183228 


1201611110 


IL 


MWU/CCOM/Adventist LaGrange Memorial Hosp 


13 


183232 


1201621109 


IL 


MWU/CCOM/Adventist Hinsdale Hospital 


14 


158101 


1201621117 


IL 


KCOM/Center Family Med/SIU Springfield Family Med 


15 


148258 


1201621365 


IL 


KCOM/SIU/Quincy 


16 


173196 


1201621467 


IL 


MWU/CCOM/Illinois Masonic Medical Ctr 


17 


158102 


1201621492 


IL 


MWU/CCOM/Carle Foundation Hospital 


18 


163821 


1201631106 


IL 


MWU/CCOM/Swedish Covenant Hospital 


19 


163823 


1201631112 


IL 


MWU/CCOMAVest Suburban Hospital Medical Center 


20 


180854 


1201721121 


IN 


PCSOM/Fort Wayne Medical Ed Program 


21 


156882 


1201921630 


KS 


Via Christi Regional Medical Center/Riverside 


22 


173195 


1202021512 


KY 


PCSOM/East Kentucky Osteo Hosp 


23 


169592 


1202211153 


ME 


UNECOM/Central Maine Medical Center 


24 


148265 


1202212152 


ME 


UNECOM/Eastern Maine Med Ctr 


25 


158105 


1202222151 


ME 


UNECOM/Maine — Dartmouth Family Practice Program 


26 


152962 


1202431159 


MA 


UNECOM/University MA (Fitchburg) Family Practice Program 


27 


162368 


1202521169 


MI 


MSUCOM/Kalamazoo Ctr Med Studies 


28 


182443 


1202521170 


MI 


EW Sparrow Hospital 


29 


169598 


1202521370 


MI 


LECOM/Marquette General Health System 


30 


126081 


1202521602 


MI 


MSUCOM/Munson Medical Center 


31 


128268 


1202531166 


MI 


Genesys Regional Med Ctr- Health Park 


32 


182483 


1202531678 


MI 


MSUCOM/Oakwood Annapolis Hospital 


33 


181016 


1202621568 


MN 


KCOM/U of Minnesota Mankato 


34 


176243 


1203311195 


NJ 


KCOM/AHS Overlook Hospital 


35 


126317 


1203321436 


NJ 


PCOMAV arren Hospital Div 


36 


137585 


1203421197 


NM 


TUCOM/University of New Mexico Hospital 
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37 


130094 


1203511203 


NY 


Wilson Memorial Reg Med Ctr 


38 


126103 


1203511206 


NY 


NYCOM/Jamaica Hospital Div 


39 


126088 


1203511207 


NY 


NYCOM/Lutheran Medical Hosp 


40 


182452 


1203511212 


NY 


NYCOM/South Nassau Communities Hosp 


41 


148266 


1203511217 


NY 


UNECOM/St Elizabeth's Hospital 


42 


180855 


1203512215 


NY 


NYCOM/Ellis Hospital McClellan Campus 


43 


169591 


1203521198 


NY 


UNECOM/Albany Medical College 


44 


126107 


1203521204 


NY 


NYCOM/The Institute for Family Health 


45 


129226 


1203521507 


NY 


NYCOM/Wyckoff Heights Medical Center 


46 


128304 


1203531681 


NJ 


St Joseph's Hospital & Med Ctr 


47 


169929 


1203621611 


NC 


PCSOM/New Hanover Regional Med. Ctr 


48 


167087 


1203631223 


NC 


NSUCOM/Duke/Southern Regional Area Health Ed Ctr 


49 


173208 


1203811234 


OH 


LECOM/Aultman Hospital 


50 


169831 


1203811237 


OH 


OUCOM/Metro-Health Medical Center 


51 


175833 


1203821231 


OH 


OUCOM/SUMMA/Akron City Hospital 


52 


157345 


1203821250 


OH 


WVSOM/The Toledo Hospital 


53 


126197 


1203831249 


OH 


OUCOM/St Vincent Mercy MC 


54 


152357 


1203921513 


OK 


OSUCOM/St Anthony Hospital 


55 


126651 


1204111260 


PA 


Altoona Hospital 


56 


162367 


1204111269 


PA 


LECOM/Conemaugh Valley Memorial Hosp 


57 


173201 


1204111277 


PA 


PCOM/Latrobe Area Hospital 


58 


163816 


1204111578 


PA 


Mercy Hospital (Pittsburgh) 


59 


169603 


1204112271 


PA 


LEC OM/UPMC McKeesport 


60 


169604 


1204112279 


PA 


LECOM/UPMC St Margaret Memorial Hospital 


61 


173203 


1204112280 


PA 


PCOM/UPMC/Shadyside Hospital 


62 


183210 


1204112283 


PA 


LECOMAV ashington Hospital 


63 


148259 


1204112579 


PA 


LECOMAV estern Pennsylvania Hospital 


64 


176235 


1204121259 


PA 


PCOM/Sacred Heart Hospital 


65 


158104 


1204121284 


PA 


PCOMAVyoming Valley Health Care 


66 


156883 


1204121518 


PA 


LECOM MedNet/Guthrie/Robert Packer Hospital 


67 


169595 


1204121572 


PA 


PCOM/Lehigh Valley Hosp Health Network 


68 


157428 


1204121603 


PA 


PCOM/St. Luke's Hospital 


69 


169599 


1204124109 


PA 


PCOM/The Medical Center Beaver PA 


70 


176183 


1204131261 


PA 


Bryn Mawr Hospital 


71 


180856 


1204131285 


PA 


PCOMAVilliamsport Hospital/Med Ctr 


72 


169602 


1204511293 


SC 


PCSOM/Spartanburg Regional Health Systems 


73 


180851 


1204821305 


TX 


Univ TX Medical Branch - Galveston - Family Practice 


74 


173200 


1204821310 


TX 


TCOM/Texas Tech Univ Hlth Sci Ctr 


75 


175836 


1204821432 


TX 


UNTHSC/TCOM/San Jacinto Methodist Hospital 


76 


163817 


1204821433 


TX 


UNTHSC/TCOM/Charlton Methodist Hospital 
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77 


152127 


1205111325 


VA 


EVVCOM/Carilion Medical Center 


78 


167091 


1205121627 


VA 


PCSOM/Shenandoah Valley Hospital 


79 


173194 


1205131323 


VA 


PCSOM/Riverside Regional Medical Ctr 


80 


182473 


1205421522 


WA 


PCSOM/Central WA Family Medicine Residency Program 


81 


152124 


1205511337 


WV 


WVSOM/Charleston Area Med Ctr 


82 


175834 


1205521335 


WV 


WVSOM/Cabell Huntington Hospital 


83 


126330 


1205522334 


WV 


WVSOM/United Hospital Center 


84 


162372 


1205522338 


WV 


W V S OM AVheeling Hospital 


85 


152130 


1205611343 


WI 


DMU/University of WI Med School 


86 


169589 


1205712351 


WY 


KCOM/University of Wyoming 
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